Systems and methods for virtual application of cosmetic products based on facial identification and corresponding makeup information

ABSTRACT

A client device obtains a request from a user to retrieve makeup information from a cloud server and obtains a digital image of a facial region of the user. The client device transmits the digital image to initiate a communication session with the cloud server without obtaining login credentials from the user, wherein the cloud server retrieves a makeup session packet by matching the transmitted digital image with one of a plurality of digital images in the cloud server and transmits the makeup session packet comprising a corresponding selection of cosmetic products. The client device receives the makeup session packet and extracts the selection of cosmetic products by the client device and generates a user interface displaying the selection of cosmetic products. The client device performs virtual application of the selection of cosmetic products onto the facial region of the user.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Divisional of pending U.S. patent application Ser.No. 16/186,853, filed on Nov. 12, 2018, which claims priority to, andthe benefit of, U.S. Provisional Patent Application entitled, “Facecombined with makeup information system,” having Ser. No. 62/733,118,filed on Sep. 19, 2018, which is incorporated by reference in itsentirety.

TECHNICAL FIELD

The present disclosure generally relates to virtual application ofcosmetic effects and more particularly, to systems and methods forperforming virtual application of cosmetic products based on facialidentification and corresponding makeup information.

BACKGROUND

Although individuals invest a substantial amount of money in makeuptools and accessories, it can be a challenge to achieve the same resultsas a makeup professional. In retail establishments, professional beautyadvisors commonly assist individuals with recommending cosmetic productsto achieve a desired look. However, due to the volume of consumersseeking makeup consultation at times, the availability of beautyadvisors may be limited. Although some retail establishments offer theuse of electronic kiosks that allow customers to evaluate cosmeticproducts, it can be difficult for customers to retain makeup informationgiven the number of cosmetic products that a customer may try on duringa particular session. Furthermore, some customers may not wish to gothrough the tedious process of setting up an account at the electronickiosk for purposes of tracking makeup information. Therefore, there is aneed for allowing consumers to efficiently evaluate and trackinformation on cosmetic products for purposes of later purchasingproducts, participating in a makeup consultation, and so on.

SUMMARY

In accordance with one embodiment, a client device obtains a requestfrom a user to retrieve makeup information from a cloud server. Theclient device obtains a digital image of a facial region of the user andtransmits the digital image to initiate a communication session with thecloud server without obtaining login credentials from the user, whereinthe cloud server retrieves a makeup session packet by matching thetransmitted digital image with one of a plurality of digital images inthe cloud server and transmits the makeup session packet comprising acorresponding selection of cosmetic products. The client device receivesthe makeup session packet and extracts the selection of cosmeticproducts. The client device generates a user interface displaying theselection of cosmetic products and performs virtual application of theselection of cosmetic products onto the facial region of the user.

Another embodiment is a system that comprises a memory storinginstructions and a processor coupled to the memory. The processor isconfigured by the instructions to obtain a request from a user toretrieve makeup information from a cloud server. The processor isfurther configured to obtain a digital image of a facial region of theuser and transmit the digital image to initiate a communication sessionwith the cloud server without obtaining login credentials from the user,wherein the cloud server retrieves a makeup session packet by matchingthe transmitted digital image with one of a plurality of digital imagesin the cloud server and transmits the makeup session packet comprising acorresponding selection of cosmetic products. The processor is furtherconfigured to receive the makeup session packet and extracting theselection of cosmetic products and generate a user interface displayingthe selection of cosmetic products. The processor is further configuredto perform virtual application of the selection of cosmetic productsonto the facial region of the user.

Other systems, methods, features, and advantages of the presentdisclosure will be or become apparent to one with skill in the art uponexamination of the following drawings and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present disclosure, and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the disclosure can be better understood withreference to the following drawings. The components in the drawings arenot necessarily to scale, with emphasis instead being placed uponclearly illustrating the principles of the present disclosure. Moreover,in the drawings, like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a block diagram of a networked environment for performingvirtual application of cosmetic products based on facial identificationand corresponding makeup information according to various embodiments ofthe present disclosure.

FIG. 2 is a schematic diagram of the devices in FIG. 1 according tovarious embodiments of the present disclosure.

FIG. 3 is a top-level flowchart illustrating examples of functionalityimplemented as portions of the devices in FIG. 1 for performing virtualapplication of cosmetic products based on facial identification andcorresponding makeup information according to various embodiments of thepresent disclosure.

DETAILED DESCRIPTION

Various embodiments are disclosed for allowing individuals to receiveand evaluate makeup product recommendations, where the individuals areable to later retrieve information relating to makeup products ofinterest through the use of a digital image, where such information maybe utilized, for example, to purchase makeup products or to participatein a makeup consultation session with an available makeup professional.Notably, various embodiments are disclosed that do not require the useof login credentials (e.g., username and password), thereby allowingindividuals to access information stored in the cloud without having tosubmit such information as contact information, e-mail address, and soon that individuals may not wish to share.

A description of a system for performing virtual application of cosmeticproducts based on facial identification and corresponding makeupinformation is now described followed by a discussion of the operationof the components within the system. FIG. 1 is a block diagram of anetworked environment that includes a makeup evaluation computing device102, a cloud server device 114, and a client device 122. The makeupevaluation computing device 102, cloud server device 114, and the clientdevice 122 are communicatively coupled via a network 112 such as, forexample, the Internet, intranets, extranets, wide area networks (WANs),local area networks (LANs), wired networks, wireless networks, or othersuitable networks, etc., or any combination of two or more suchnetworks.

Both the makeup evaluation computing device 102 and the client device122 may be embodied as a computing device such as, but not limited to, asmartphone, a tablet computing device, a laptop, and so on. The cloudserver device 114 may comprise a server computer or any other systemproviding computing capability. Alternatively, the cloud server device114 may employ a plurality of computing devices that can be arranged,for example, in one or more server banks, computer banks or otherarrangements. Such computing devices can be located in a singleinstallation or can be distributed among different geographicallocations.

The makeup evaluation computing device 102 may be installed in a retailestablishment and is utilized by a user to receive and evaluate makeupproduct recommendations generated by the makeup evaluation computingdevice 102. A makeup evaluation service 104 executes on a processor ofthe makeup evaluation computing device 102 thereby causing the makeupevaluation computing device 102 to perform the operations/functions forimplementing the features disclosed herein. The makeup evaluationservice 104 includes a session manager 106, a user interface (UI)generator 108, a virtual makeup applicator 110, and a packet generator111.

The session manager 106 is configured to obtain a request from a user toinitiate a makeup evaluation session. The session manager 106 initiatesthe makeup evaluation session without obtaining login credentials fromthe user. Rather than utilizing traditional login credentials (e.g.,username and password) to initiate the makeup session, the sessionmanager 106 obtains a digital image of a facial region of the user andutilizes the digital image to initiate the makeup evaluation session.One advantage of bypassing the use of login credentials is that the userdoes not have to spend time setting up an account on the makeupevaluation computing device 102 to access content on the cloud serverdevice 114. Furthermore, the user does not have to submit suchinformation as contact information, e-mail address, and so on thatindividuals may wish to keep private.

As one of ordinary skill will appreciate, the digital image may beencoded in any of a number of formats including, but not limited to,JPEG (Joint Photographic Experts Group) files, TIFF (Tagged Image FileFormat) files, PNG (Portable Network Graphics) files, GIF (GraphicsInterchange Format) files, BMP (bitmap) files or any number of otherdigital formats. Alternatively, the digital image may be derived from astill image of a video encoded in formats including, but not limited to,Motion Picture Experts Group (MPEG)-1, MPEG-2, MPEG-4, H.264, ThirdGeneration Partnership Project (3GPP), 3GPP-2, Standard-Definition Video(SD-Video), High-Definition Video (HD-Video), Digital Versatile Disc(DVD) multimedia, Video Compact Disc (VCD) multimedia, High-DefinitionDigital Versatile Disc (HD-DVD) multimedia, Digital TelevisionVideo/High-definition Digital Television (DTV/HDTV) multimedia, AudioVideo Interleave (AVI), Digital Video (DV), QuickTime (QT) file, WindowsMedia Video (WMV), Advanced System Format (ASF), Real Media (RM), FlashMedia (FLV), an MPEG Audio Layer III (MP3), an MPEG Audio Layer II(MP2), Waveform Audio Format (WAV), Windows Media Audio (WMA), 360degree video, 3D scan model, or any number of other digital formats.

The UI generator 108 is configured to generate a user interfacedisplaying one or more cosmetic products. The UI generator 108 obtainsfrom the user a selection of cosmetic products among the one or moredisplayed cosmetic products. The virtual makeup applicator 110 isconfigured to perform virtual application of the selected cosmeticproducts on the facial region of the user, thereby allowing the user toevaluate the application of the selected cosmetic products. The packetgenerator 111 is configured to generate a makeup session packet 113comprising the digital image 109 of the facial region of the user andthe stored selection of cosmetic products 115. The packet generator 111transmits the makeup session packet 113 to the cloud server device 114.A data management service 116 executes on a processor of the cloudserver device 114 and stores the makeup session packet 113 in a datastore 118.

A virtual makeup application 124 executes on a processor of the clientdevice 122 thereby causing the client device 122 to perform theoperations/functions for implementing the features disclosed herein. Thevirtual makeup applicator 124 is configured to obtain a request from auser to retrieve makeup information from the cloud server device 114.The virtual makeup application 124 initiates a communication sessionwith the cloud server device 114 without obtaining login credentialsfrom the user. As discussed above, one advantage of bypassing the use oflogin credentials is that the user does not have to spend time settingup an account to access content on the cloud server device 114.Furthermore, the user does not have to submit such information ascontact information, e-mail address, and so on that individuals may wishto keep private.

The virtual makeup application 124 obtains a digital image of a facialregion of the user and transmits the digital image to the cloud serverdevice 114, wherein the cloud server device 114 retrieves a makeupsession packet 113 comprising a matching digital image of the facialregion of the user and a corresponding selection of cosmetic products.The cloud server device 114 sends the makeup session packet 113 to theclient device 122, and the virtual makeup application 124 generates auser interface displaying the selection of cosmetic products specifiedin the makeup session packet 113. The virtual makeup application 124performs virtual application of the selection of cosmetic products ontothe facial region of the user.

FIG. 2 illustrates a schematic block diagram for each of the makeupevaluation computing device 102, the cloud server device 114, and theclient device 122 in FIG. 1. Each of these computing devices 102, 114,122 may be embodied in any one of a wide variety of wired and/orwireless computing devices, such as a desktop computer, portablecomputer, dedicated server computer, multiprocessor computing device,smart phone, tablet, and so forth. As shown in FIG. 2, each of thesecomputing devices 102, 114, 122 comprises memory 214, a processingdevice 202, a number of input/output interfaces 204, a network interface206, a display 208, a peripheral interface 211, and mass storage 226,wherein each of these components are connected across a local data bus210.

The processing device 202 may include any custom made or commerciallyavailable processor, a central processing unit (CPU) or an auxiliaryprocessor among several processors associated with the computing device,a semiconductor based microprocessor (in the form of a microchip), amacroprocessor, one or more application specific integrated circuits(ASICs), a plurality of suitably configured digital logic gates, andother well known electrical configurations comprising discrete elementsboth individually and in various combinations to coordinate the overalloperation of the computing system.

The memory 214 may include any one of a combination of volatile memoryelements (e.g., random-access memory (RAM, such as DRAM, and SRAM,etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape,CDROM, etc.). The memory 214 typically comprises a native operatingsystem 216, one or more native applications, emulation systems, oremulated applications for any of a variety of operating systems and/oremulated hardware platforms, emulated operating systems, etc. Forexample, the applications may include application specific softwarewhich may comprise some or all the components of the computing devices102, 114, 122 depicted in FIG. 1. In accordance with such embodiments,the components are stored in memory 214 and executed by the processingdevice 202, thereby causing the processing device 202 to perform theoperations/functions relating to the features disclosed herein. One ofordinary skill in the art will appreciate that the memory 214 can, andtypically will, comprise other components which have been omitted forpurposes of brevity. For some embodiments, the components in thecomputing devices 102, 114, 122 may be implemented by hardware and/orsoftware.

Input/output interfaces 204 provide any number of interfaces for theinput and output of data. For example, where the computing devicecomprises a personal computer, these components may interface with oneor more user input/output interfaces 204, which may comprise a keyboardor a mouse, as shown in FIG. 2. The display 208 may comprise a computermonitor, a plasma screen for a PC, a liquid crystal display (LCD) on ahand held device, a touchscreen, or other display device.

In the context of this disclosure, a non-transitory computer-readablemedium stores programs for use by or in connection with an instructionexecution system, apparatus, or device. More specific examples of acomputer-readable medium may include by way of example and withoutlimitation: a portable computer diskette, a random access memory (RAM),a read-only memory (ROM), an erasable programmable read-only memory(EPROM, EEPROM, or Flash memory), and a portable compact disc read-onlymemory (CDROM) (optical).

Reference is made to FIG. 3, which is a flowchart 300 that shows theinteraction between the various components in the networked environmentof FIG. 1 for performing virtual application of cosmetic products basedon facial identification and corresponding makeup information. It isunderstood that the flowchart 300 of FIG. 3 provides merely an exampleof the different types of functional arrangements that may be employedto implement the operation of the various components of the computingdevices 102, 114, 122 depicted in FIG. 1. As an alternative, theflowchart 300 of FIG. 3 may be viewed as depicting an example of stepsof a method implemented in the computing devices 102, 114, 122 accordingto one or more embodiments.

Although the flowchart 300 of FIG. 3 shows a specific order ofexecution, it is understood that the order of execution may differ fromthat which is depicted. For example, the order of execution of two ormore blocks may be scrambled relative to the order shown. Also, two ormore blocks shown in succession in FIG. 3 may be executed concurrentlyor with partial concurrence. It is understood that all such variationsare within the scope of the present disclosure.

At block 310, the makeup evaluation computing device 102 initiates amakeup evaluation session based on input from a user. For someembodiments, the makeup evaluation computing device 102 may be installedin a retail establishment. At block 315, the makeup evaluation computingdevice 102 obtains a digital image of the facial region of the user andprovides cosmetic products. For some embodiments, the cosmetic productsare displayed on a user interface generated by the makeup evaluationcomputing device 102, where the cosmetic products may be displayed basedon facial features and attributes depicted in the digital image. Atblock 320, the makeup evaluation computing device 102 obtains selectionsof one or more of the cosmetic products and performs virtual applicationof the selected cosmetic products onto the digital image of the user.

At block 325, the makeup evaluation computing device 102 generates amakeup session packet 113 (FIG. 1) by bundling the digital image withthe cosmetic products selected by the user. Other information such asprice, ingredients, manufacturing location, usage about the cosmeticproducts, and so on may also be bundled into the makeup session packet.The makeup evaluation computing device 102 then uploads the makeupsession packet to the cloud server device 114. At block 330, the cloudserver device 114 stores the makeup session packet 113 in a data store118 (FIG. 1).

At block 335, the makeup evaluation computing device 102 terminates themakeup evaluation session. The termination process may comprise deletingthe digital image of the user and the stored selection of cosmeticproducts selected by the user from a local storage of the makeupevaluation computing device 102. In this regard, the digital image andthe selection of cosmetic products are only stored temporarily by themakeup evaluation computing device 102.

Later, the user may wish to retrieve information relating to thecosmetic products selected earlier by the user while visiting the retailestablishment. In exemplary embodiments, the user accesses thisinformation using a client device 122 such as the user's smartphone,laptop, or other computing device communicatively coupled to the cloudserver device 114. At block 340, the client device 122 obtains a digitalimage of the facial region of the user. At block 345, the client device122 transmits the digital image to the cloud server device 114 toinitiate a communication session with the cloud server device 114. Notethat the client device 122 initiates the communication session withoutthe use of any kind of login credentials (e.g., username and password)associated with the user.

At block 350, the cloud server device 114 analyzes the digital imageuploaded by the client device 122 and searches the data store 118 andcompares the uploaded digital image to digital images in each of themakeup session packets 113 in the data store 118. The cloud serverdevice 114 retrieves a closest matching digital image in the data store118 and transmits the corresponding makeup session packet 113 to theclient device 122.

At block 355, the client device 122 parses the makeup session packet 113downloaded from the cloud server device 114 and obtains the cosmeticproducts embedded in the makeup session packet 113. For someembodiments, the client device 122 generates a user interface thatdisplays a listing of the cosmetic products in the order they wereselected earlier by the user using the makeup evaluation computingdevice 102 at the retail establishment. The client device 122 performsvirtual application of the cosmetic products obtained from thedownloaded the makeup session packet 113.

For some embodiments, the client device 122 may also determining agrouping of beauty advisors currently online and display a listing ofbeauty advisors based on the cosmetic products specified in the makeupsession packet 113 downloaded from the cloud server device 114.Specifically, the client device 122 may be configured to retrieveprofile information for each of the online beauty advisors and identifybeauty advisors who have prior experience working with one or more ofthe cosmetic products specified in the makeup session packet 113. Theclient device 122 displays a list of suitable beauty advisors to theuser (block 360).

The user may then initiate a makeup consultation session with a beautyadvisor to receive further cosmetic products or to further refineapplication of the cosmetic products previously selected by the user.For some embodiments, the user may be prompted to enter logincredentials by the client device 122 in order to provide the beautyadvisor with relevant information about the user for purposes offacilitating the makeup consultation session. Additional cosmeticproducts and refinement of the previously selected cosmetic productsmade during the makeup consultation session may then be stored inconnection with the user's login credentials. Thereafter, the process inFIG. 3 ends.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations setforth for a clear understanding of the principles of the disclosure.Many variations and modifications may be made to the above-describedembodiment(s) without departing substantially from the spirit andprinciples of the disclosure. All such modifications and variations areintended to be included herein within the scope of this disclosure andprotected by the following claims.

At least the following is claimed:
 1. A method implemented in a clientdevice, comprising: obtaining a request from a user to retrieve makeupinformation from a cloud server; obtaining a digital image of a facialregion of the user; transmitting the digital image to initiate acommunication session with the cloud server without obtaining logincredentials from the user, wherein the cloud server retrieves a makeupsession packet by matching the transmitted digital image with one of aplurality of digital images in the cloud server and transmits the makeupsession packet comprising a corresponding selection of cosmeticproducts; receiving the makeup session packet and extracting theselection of cosmetic products by the client device; generating a userinterface displaying the selection of cosmetic products; and performingvirtual application of the selection of cosmetic products onto thefacial region of the user.
 2. The method of claim 1, further comprising:determining a grouping of beauty advisors currently online; generating asubset of beauty advisors from the grouping based on the selection ofcosmetic products; displaying the subset of beauty advisors in the userinterface; obtaining a selection from the user of the of the beautyadvisors; and establishing a makeup consultation session between theclient device and a device of the selected beauty advisor.
 3. The methodof claim 2, wherein generating the subset of beauty advisors from thegrouping based on the selection of cosmetic products comprises:retrieving profile information for each of the grouping of beautyadvisors; and identifying beauty advisors within the grouping of beautyadvisors with prior experience applying at least one of the selection ofcosmetic products based on the profile information.
 4. The method ofclaim 2, wherein establishing the makeup consultation session betweenthe client device and a device of the selected beauty advisor comprises:obtaining login credentials from the user; and establishing the makeupconsultation session between the client device and a device of theselected beauty advisor based on the login credentials.
 5. The method ofclaim 1, wherein the selection of cosmetic products are displayed in asame order in which the user previously evaluated the cosmetic productsat a retail establishment using a makeup evaluation computing device isinstalled in the retail establishment.
 6. The method of claim 1, furthercomprising displaying in the user interface an ordered listing of asequence of cosmetic products selected by the user of the client device.7. A system, comprising: a memory storing instructions; a processorcoupled to the memory and configured by the instructions to at least:obtain a request from a user to retrieve makeup information from a cloudserver; obtain a digital image of a facial region of the user; transmitthe digital image to initiate a communication session with the cloudserver without obtaining login credentials from the user, wherein thecloud server retrieves a makeup session packet by matching thetransmitted digital image with one of a plurality of digital images inthe cloud server and transmits the makeup session packet comprising acorresponding selection of cosmetic products; receive the makeup sessionpacket and extract the selection of cosmetic products; generate a userinterface displaying the selection of cosmetic products; and performvirtual application of the selection of cosmetic products onto thefacial region of the user.
 8. The system of claim 7, wherein theprocessor is further configured to: determine a grouping of beautyadvisors currently online; generate a subset of beauty advisors from thegrouping based on the selection of cosmetic products; display the subsetof beauty advisors in the user interface; obtain a selection from theuser of the of the beauty advisors; and establish a makeup consultationsession with a device of the selected beauty advisor.
 9. The system ofclaim 8, wherein the processor generates the subset of beauty advisorsfrom the grouping based on the selection of cosmetic products byperforming the steps of: retrieving profile information for each of thegrouping of beauty advisors; and identifying beauty advisors within thegrouping of beauty advisors with prior experience applying at least oneof the selection of cosmetic products based on the profile information.10. The system of claim 8, wherein the processor establishes the makeupconsultation session with the device of the selected beauty advisor byperforming the steps of: obtaining login credentials from the user; andestablishing the makeup consultation session with the device of theselected beauty advisor based on the login credentials.
 11. The systemof claim 7, wherein the selection of cosmetic products are displayed ina same order in which the user previously evaluated the cosmeticproducts at a retail establishment using a makeup evaluation computingdevice is installed in the retail establishment.
 12. The system of claim7, wherein the processor is further configured to display in the userinterface an ordered listing of a sequence of cosmetic products selectedby the user of the system.